Proxmox Virtual Environment (Proxmox VE or PVE) is an

open-source software Open-source software (OSS) is computer software that is released under a license in which the copyright holder grants users the rights to use, study, change, and distribute the software and its source code to anyone and for any purpose. Op ...

server for

virtualization In computing, virtualization or virtualisation (sometimes abbreviated v12n, a numeronym) is the act of creating a virtual (rather than actual) version of something at the same abstraction level, including virtual computer hardware platforms, stor ...

management. It is a hosted

hypervisor A hypervisor (also known as a virtual machine monitor, VMM, or virtualizer) is a type of computer software, firmware or hardware that creates and runs virtual machines. A computer on which a hypervisor runs one or more virtual machines is called ...

that can run operating systems including Linux and Windows on x64 hardware. It is a Debian-based Linux distribution with a modified

Ubuntu Ubuntu ( ) is a Linux distribution based on Debian and composed mostly of free and open-source software. Ubuntu is officially released in three editions: '' Desktop'', ''Server'', and ''Core'' for Internet of things devices and robots. All ...

LTS kernel and allows deployment and management of

virtual machine In computing, a virtual machine (VM) is the virtualization/ emulation of a computer system. Virtual machines are based on computer architectures and provide functionality of a physical computer. Their implementations may involve specialized h ...

s and

containers A container is any receptacle or enclosure for holding a product used in storage, packaging, and transportation, including shipping. Things kept inside of a container are protected on several sides by being inside of its structure. The term ...

. Proxmox VE includes a web console and command-line tools, and provides a

REST Rest or REST may refer to: Relief from activity * Sleep ** Bed rest * Kneeling * Lying (position) * Sitting * Squatting position Structural support * Structural support ** Rest (cue sports) ** Armrest ** Headrest ** Footrest Arts and enter ...

API for third-party tools. Two types of virtualization are supported: container-based with LXC (starting from version 4.0 replacing

OpenVZ OpenVZ (Open Virtuozzo) is an operating-system-level virtualization technology for Linux. It allows a physical server to run multiple isolated operating system instances, called containers, virtual private servers (VPSs), or virtual environments ...

used in version up to 3.4, included), and full virtualization with KVM. It includes a web-based management interface. Proxmox VE is licensed under the

GNU Affero General Public License The GNU Affero General Public License (GNU AGPL) is a free, copyleft license published by the Free Software Foundation in November 2007, and based on the GNU General Public License, version 3 and the Affero General Public License. The Free So ...

, version 3.

History

Development of Proxmox VE started when Dietmar Maurer and Martin Maurer, two Linux developers, found out

had no backup tool and no management GUI. KVM was appearing at the same time in Linux, and was added shortly afterwards. The first public release took place in April 2008. It supported container and full virtualization, managed with a web-based user interface similar to other commercial offerings.

Features

Proxmox VE is an open-source server virtualization platform to manage two virtualization technologies:

Kernel-based Virtual Machine Kernel-based Virtual Machine (KVM) is a virtualization module in the Linux kernel that allows the kernel to function as a hypervisor. It was merged into the mainline Linux kernel in version 2.6.20, which was released on February 5, 2007. KV ...

(KVM) for virtual machines and LXC for containers - with a single web-based interface. The source is open, based on the GNU AGPL, v3. The company sells optional subscription-based

customer support Customer support is a range of services to assist customers in making cost effective and correct use of a product. It includes assistance in planning, installation, training, troubleshooting, maintenance, upgrading, and disposal of a product. Reg ...

. With a subscription, users get access to an enterprise

software repository A software repository, or repo for short, is a storage location for software packages. Often a table of contents is also stored, along with metadata. A software repository is typically managed by source control or repository managers. Package ...

VM emulated/paravirtualized hardware

Since Proxmox VE for Virtual Machines are using KVM (+

QEMU QEMU is a free and open-source emulator (Quick EMUlator). It emulates the machine's central processing unit, processor through dynamic binary translation and provides a set of different hardware and device models for the machine, enabling it t ...

) as hypervisor, it natively (from the GUI) supports many emulated and paravirtualized hardware components such as: * Motherboard chipset: i440fx or q35 (many versions) * BIOS: SeaBIOS or OVMF (

UEFI UEFI (Unified Extensible Firmware Interface) is a set of specifications written by the UEFI Forum. They define the architecture of the platform firmware used for booting and its interface for interaction with the operating system. Examples of ...

) *

Processor Processor may refer to: Computing Hardware * Processor (computing) **Central processing unit (CPU), the hardware within a computer that executes a program *** Microprocessor, a central processing unit contained on a single integrated circuit (I ...

types: host, kvm32, kvm64, qemu32, qemu64 or specific CPU generation (features/flags) types like: athlon, opteron, phenom, broadwell, coreduo * Disk controllers: LSI53C895A, LSI53C810, MegaRAID SAS 8708EI, VirtIO SCSI (paravirtualized), VirtIO SCSI Single (paravirtualized), VMware PVSCSI * Display: Standard

VGA Video Graphics Array (VGA) is a video display controller and accompanying de facto graphics standard, first introduced with the IBM PS/2 line of computers in 1987, which became ubiquitous in the PC industry within three years. The term can no ...

VMware VMware, Inc. is an American cloud computing and virtualization technology company with headquarters in Palo Alto, California. VMware was the first commercially successful company to virtualize the x86 architecture. VMware's desktop software ru ...

compatible, SPICE, Serial, VirtIO GPU (paravirtualized), VirGL GPU (paravirtualized) *

Network Network, networking and networked may refer to: Science and technology * Network theory, the study of graphs as a representation of relations between discrete objects * Network science, an academic field that studies complex networks Mathematics ...

: Intel E1000, VirtIO (paravirtualized), Realtek 8139, VMware vmxnet3 * RNG: VirtIO RNG * Balloon capabilities (VirtIO) * Support for QEMU guest agent. Proxmox VE also supports PCI and

USB Universal Serial Bus (USB) is an industry standard that establishes specifications for cables, connectors and protocols for connection, communication and power supply (interfacing) between computers, peripherals and other computers. A broad ...

pass-through devices from the physical server. Additional options like CPU flags, different type of emulated hardware (that QEMU supports) etc. are also available from the CLI (shell).

Supported VM/Linux container operating systems

Since Proxmox VE is using KVM in the background, it supports all operating systems that KVM supports for virtual machines, like: *

Linux Linux ( or ) is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically packaged as a Linux distribution, w ...

: any distribution (

Red Hat Enterprise Linux Red Hat Enterprise Linux (RHEL) is a commercial open-source Linux distribution developed by Red Hat for the commercial market. Red Hat Enterprise Linux is released in server versions for x86-64, Power ISA, ARM64, and IBM Z and a desktop ...

, CentOS,

Rocky Linux Rocky Linux is a Linux distribution developed by Rocky Enterprise Software Foundation, which is a privately owned benefit corporation that describes itself as a "self imposed not-for-profit". It is intended to be a downstream, complete binary-co ...

, Fedora,

Oracle Enterprise Linux Oracle Linux (abbreviated OL, formerly known as Oracle Enterprise Linux or OEL) is a Linux distribution packaged and freely distributed by Oracle, available partially under the GNU General Public License since late 2006. It is compiled from Red ...

, Debian,

, SUSE Linux Enterprise Server,

openSUSE openSUSE () is a free and open source RPM-based Linux distribution developed by the openSUSE project. The initial release of the community project was a beta version of SUSE Linux 10.0. Additionally the project creates a variety of tools, s ...

, Gentoo,

Arch Linux Arch Linux () is an independently developed, x86-64 general-purpose Linux distribution that strives to provide the latest stable versions of most software by following a rolling-release model. The default installation is a minimal base system, ...

, ... ). * BSD: FreeBSD, OpenBSD, NetBSD,

DragonflyBSD DragonFly BSD is a free and open-source Unix-like operating system forked from FreeBSD 4.8. Matthew Dillon, an Amiga developer in the late 1980s and early 1990s and FreeBSD developer between 1994 and 2003, began working on DragonFly BSD in ...

, ... * Solaris (like): Solaris, OpenSolaris,

OpenIndiana OpenIndiana is a free and open-source illumos distribution Unix-derived System V SVR4 operating system, started as OpenSolaris continuation project. Forked from OpenSolaris after OpenSolaris was discontinued by Oracle Corporation, OpenIndiana ...

, Nexenta Core, ... * Windows: almost any version starting from Windows 95 (

Windows 95 Windows 95 is a consumer-oriented operating system developed by Microsoft as part of its Windows 9x family of operating systems. The first operating system in the 9x family, it is the successor to Windows 3.1x, and was released to manufacturi ...

Windows NT Windows NT is a proprietary graphical operating system produced by Microsoft, the first version of which was released on July 27, 1993. It is a processor-independent, multiprocessing and multi-user operating system. The first version of Win ...

Windows 98SE Windows is a group of several Proprietary software, proprietary graphical user interface, graphical operating system families developed and marketed by Microsoft. Each family caters to a certain sector of the computing industry. For example, W ...

, Windows Server 2000,

Windows 2000 Windows 2000 is a major release of the Windows NT operating system developed by Microsoft and oriented towards businesses. It was the direct successor to Windows NT 4.0, and was released to manufacturing on December 15, 1999, and was officiall ...

Windows XP Windows XP is a major release of Microsoft's Windows NT operating system. It was released to manufacturing on August 24, 2001, and later to retail on October 25, 2001. It is a direct upgrade to its predecessors, Windows 2000 for high-end and ...

Windows Server 2003 Windows Server 2003 is the sixth version of Windows Server operating system produced by Microsoft. It is part of the Windows NT family of operating systems and was released to manufacturing on March 28, 2003 and generally available on April 24, 2 ...

/ R2,

Windows Vista Windows Vista is a major release of the Windows NT operating system developed by Microsoft. It was the direct successor to Windows XP, which was released five years before, at the time being the longest time span between successive releases of ...

Windows Server 2008 Windows Server 2008 is the fourth release of the Windows Server operating system produced by Microsoft as part of the Windows NT family of the operating systems. It was released to manufacturing on February 4, 2008, and generally to retail on F ...

/ R2,

Windows 7 Windows 7 is a major release of the Windows NT operating system developed by Microsoft. It was released to manufacturing on July 22, 2009, and became generally available on October 22, 2009. It is the successor to Windows Vista, released nearly ...

/ 8,

Windows 10 Windows 10 is a major release of Microsoft's Windows NT operating system. It is the direct successor to Windows 8.1, which was released nearly two years earlier. It was released to manufacturing on July 15, 2015, and later to retail on J ...

, Windows Server 2008R2,

Windows Server 2012 Windows Server 2012, codenamed "Windows Server 8", is the sixth version of the Windows Server operating system by Microsoft, as part of the Windows NT family of operating systems. It is the server version of Windows based on Windows 8 and succe ...

/ R2,

Windows 11 Windows 11 is the latest major release of Microsoft's Windows NT operating system, released in October 2021. It is a free upgrade to its predecessor, Windows 10 (2015), and is available for any Windows 10 devices that meet the new Windows 11 ...

). *

Haiku is a type of short form poetry originally from Japan. Traditional Japanese haiku consist of three phrases that contain a ''kireji'', or "cutting word", 17 '' on'' (phonetic units similar to syllables) in a 5, 7, 5 pattern, and a ''kigo'', or s ...

, ReactOS, Plan 9,

AROS Research Operating System AROS Research Operating System (AROS, pronounced "AR-OS") is a free and open-source software, free and open-source multi media centric implementation of the AmigaOS 3.1 application programming interface (API). Designed to be portable and flexib ...

and

macOS macOS (; previously OS X and originally Mac OS X) is a Unix operating system developed and marketed by Apple Inc. since 2001. It is the primary operating system for Apple's Mac computers. Within the market of desktop and lapt ...

. * In addition it supports: Android, GNU/Hurd ( Debian K16), Minix 3.1.2a, together with other operating systems. Relating to Linux containers, since Proxmox VE is using LXC Linux containers, it supports any Linux distribution as a Linux container.

Central management

Proxmox VE use integrated web based management based on

cluster may refer to: Science and technology Astronomy * Cluster (spacecraft), constellation of four European Space Agency spacecraft * Asteroid cluster, a small asteroid family * Cluster II (spacecraft), a European Space Agency mission to study t ...

and Proxmox VE cluster

filesystem In computing, file system or filesystem (often abbreviated to fs) is a method and data structure that the operating system uses to control how data is Computer data storage, stored and retrieved. Without a file system, data placed in a storage me ...

(pmxcfs). It means that there is no

single point of failure A single point of failure (SPOF) is a part of a system that, if it fails, will stop the entire system from working. SPOFs are undesirable in any system with a goal of high availability or reliability, be it a business practice, software appl ...

in cluster, since every

node In general, a node is a localized swelling (a "knot") or a point of intersection (a vertex). Node may refer to: In mathematics * Vertex (graph theory), a vertex in a mathematical graph *Vertex (geometry), a point where two or more curves, lines ...

(physical host) has all the configuration files stored in database-driver multi master cluster wide filesystem, shared among the nodes via

corosync The Corosync Cluster Engine is an open source implementation of the Totem Single Ring Ordering and Membership protocol. It was originally derived from the OpenAIS project and licensed under the new BSD License. The mission of the Corosync effo ...

. Even more, every node has all the roles/functions (management/firewall/storage/network/...) needed for work. As the cluster grows, level of redundancy also grows with them. It also means that user can configure anything on any node inside the cluster, since all the changes are synchronized inside the cluster and immediately available to the other nodes.

Network model

Proxmox VE use

bridge A bridge is a structure built to span a physical obstacle (such as a body of water, valley, road, or rail) without blocking the way underneath. It is constructed for the purpose of providing passage over the obstacle, which is usually somethi ...

model by default. Bridges are like physical network switches that work in

OSI OSI may refer to: Places * Osijek Airport (IATA code: OSI), an airport in Croatia * Ősi, a village in Veszprém county, Hungary * Oši, an archaeological site in Semigallia, Latvia * Osi, a village in Ido-Osi, Ekiti State, Nigeria * Osi, Ekiti ...

layer 2 (OSI 2), but implemented in software. All virtual machines or Linux containers can share a single bridge, or you can create multiple bridges to isolate (segment) the network. For the bridge itself you can choose between Linux native bridge and/or

Open vSwitch Open vSwitch, sometimes abbreviated as OVS, is an open-source implementation of a distributed virtual multilayer switch. The main purpose of Open vSwitch is to provide a switching stack for hardware virtualization environments, while supporti ...

bridge. In addition to that you can also natively use a

bond Bond or bonds may refer to: Common meanings * Bond (finance), a type of debt security * Bail bond, a commercial third-party guarantor of surety bonds in the United States * Chemical bond, the attraction of atoms, ions or molecules to form chemica ...

(aggregation): Linux native or Open vSwitch kind, with the mode that is best suited for you (e.g. balance-rr, active-backup, balance-xor, broadcast, LACP, balance-tlb, balance-alb). All of the above interfaces support VLANs (

802.1Q IEEE 802.1Q, often referred to as Dot1q, is the networking standard that supports virtual local area networking (VLANs) on an IEEE 802.3 Ethernet network. The standard defines a system of VLAN tagging for Ethernet frames and the accompanying pro ...

). There is an additional SDN module (currently in testing in v.7.2.x) with which you can utilize:

VLAN A virtual local area network (VLAN) is any broadcast domain that is partitioned and isolated in a computer network at the data link layer ( OSI layer 2).IEEE 802.1Q-2011, ''1.4 VLAN aims and benefits'' In this context, virtual, refers to a ph ...

, QinQ,

VxLAN Virtual Extensible LAN (VXLAN) is a network virtualization technology that attempts to address the scalability problems associated with large cloud computing deployments. It uses a VLAN-like encapsulation technique to encapsulate OSI layer 2 Eth ...

and

EVPN Ethernet VPN (EVPN) is a technology for carrying layer 2 Ethernet traffic as a virtual private network using wide area network protocols. EVPN technologies include Ethernet over MPLS and Ethernet over VXLAN. EVPNs are covered by a number of Intern ...

for network isolation/segmentation, among the other options specific to the SDN.

Storage model

Proxmox VE supports flexible local and remote (network) storages. Supported local storage types are: * LVM group, LVM-Thin, directory,

ZFS ZFS (previously: Zettabyte File System) is a file system with volume management capabilities. It began as part of the Sun Microsystems Solaris operating system in 2001. Large parts of Solaris – including ZFS – were published under an ope ...

and

Btrfs Btrfs (pronounced as "better F S", "butter F S", "b-tree F S", or simply by spelling it out) is a computer storage format that combines a file system based on the copy-on-write (COW) principle with a logical volume manager (not to be confused ...

. Supported remote (network) storage types are: *

iSCSI Internet Small Computer Systems Interface or iSCSI ( ) is an Internet Protocol-based storage networking standard for linking data storage facilities. iSCSI provides block-level access to storage devices by carrying SCSI commands over a TCP/IP ...

, Fibre Channel, NFS,

GlusterFS Gluster Inc. (formerly known as Z RESEARCH) was a software company that provided an open source platform for scale-out public and private cloud storage. The company was privately funded and headquartered in Sunnyvale, California, with an enginee ...

SMB/CIFS Server Message Block (SMB) is a communication protocol originally developed in 1983 by Barry A. Feigenbaum at IBM and intended to provide shared access to files and printers across nodes on a network of systems running IBM's OS/2. It also provides ...

, ZFS over iSCSI, Ceph ( Ceph FS and/or

RBD RBD is a Mexican Latin pop group that gained popularity from Televisa's telenovela ''Rebelde''. The group achieved international success from 2004 until their separation in 2009 and sold over 15 million records worldwide, making them one of ...

) and

Proxmox Backup Server Proxmox Backup Server (short Proxmox BS) is an open-source backup software project supporting virtual machines, containers, and physical hosts. The Bare-metal server is based on the Debian Linux distribution, with some extended features, suc ...

.
In addition, storages can be divided into two groups: * File level storage - which allow access to fully featured POSIX compliant filesystem. * Block level storage - which allow access to raw (block) data. Even more, there is a need (and possibilities) to define the type of the storage (for each and every storage), depending on the usage of the specific storage (for storing the VM/Container disk images, ISO images, LXC templates, backups, etc).

Storage replication

Proxmox VE supports redundancy for guests that are using the local storage in the way that it can replicate guest volumes to another PVE node (host) using the method so called storage replication (available only for

). Those replication tasks can be started automatically in selected replication time interval.

High-availability cluster

Proxmox VE (PVE) can be clustered across multiple server nodes. Official documentation guarantees/tested the cluster size of up to 32 physical servers (nodes), no matter the CPU core number or any other hardware limitations, but even bigger cluster size is possible. Those limitation are not imposed by PVE itself, but come from

Corosync The Corosync Cluster Engine is an open source implementation of the Totem Single Ring Ordering and Membership protocol. It was originally derived from the OpenAIS project and licensed under the new BSD License. The mission of the Corosync effo ...

limits. Since version 2.0, Proxmox VE offers a

high availability High availability (HA) is a characteristic of a system which aims to ensure an agreed level of operational performance, usually uptime, for a higher than normal period. Modernization has resulted in an increased reliance on these systems. F ...

option for clusters based on the Corosync communication stack. Starting from the PVE v.6.0 Corosync v.3.x is in use (not compatible with the earlier PVE versions). Individual virtual servers can be configured for high availability, using the built-in ha-manager. If a Proxmox node becomes unavailable or fails, the virtual servers can be automatically moved to another node and restarted. The database and FUSE-based Proxmox Cluster filesystem (pmxcfs) makes it possible to perform the configuration of each cluster node via the Corosync communication stack with

SQlite SQLite (, ) is a database engine written in the C programming language. It is not a standalone app; rather, it is a library that software developers embed in their apps. As such, it belongs to the family of embedded databases. It is the m ...

engine.

VM/containers migration

At least since 2012, in a HA cluster, live virtual machines can be moved from one physical host to another without downtime. Since Proxmox VE 1.0, released 29.10.2008 KVM and

live migration is supported (starting from version 4.0

is replaced by LXC). Related to migration, PVE support next VM/Container migration types: * Offline * Online (with some limitations related to LXC containers) Also PVE supports VM/Container migration from local storage's (local to Host/Hypervisor) or from (using) a shared storages (e.g. NFS/iSCSI/Ceph).

Clones and templates

PVE supports clones and templates. Clones are exact copy of the VM/container and a template can be created from VM/Container. One can use the template(s) to clone them to the new VM/container.

Virtual appliances

Proxmox VE has pre-packaged server

software appliances A software appliance is a software application combined with just enough operating system (JeOS) to run optimally on industry-standard hardware (typically a server) or in a virtual machine. It is a software distribution or firmware that impleme ...

which can be downloaded via the GUI. It is possible to download and deploy appliances from the

TurnKey Linux Virtual Appliance Library The TurnKey Linux Virtual Appliance Library is a free open-source software project which develops a range of Debian-based pre-packaged server software appliances (also called virtual appliances). Turnkey appliances can be deployed as a virtual ma ...

Data backup

PVE includes backup software, vzdump, which allows

data compression In information theory, data compression, source coding, or bit-rate reduction is the process of encoding information using fewer bits than the original representation. Any particular compression is either lossy or lossless. Lossless compressio ...

and on-line operation (snapshot mode). Since 2020 there's also an advanced client-server software for that called

(PBS), which offers deduplication, compression,

authenticated encryption Authenticated Encryption (AE) and Authenticated Encryption with Associated Data (AEAD) are forms of encryption which simultaneously assure the confidentiality and authenticity of data. Programming interface A typical programming interface for ...

and

incremental backups An incremental backup is one in which successive copies of the data contain only the portion that has changed since the preceding backup copy was made. When a full recovery is needed, the restoration process would need the last full backup plus al ...

. Backup can be made only on storages selected as a backup (type) of the storage, and they may be on: * NFS, SMB/CIFS, iSCSI, Ceph RBD, Proxmox Backup Server (even on local disk) There are a few levels of backup: * Scheduled Backup job from the cluster view on selected VM/containers to selected backup storage * Manual backup of only selected VM/container There are also a few backup modes: * Stop mode - provides highest consistency, it includes stopping VM/container, creating backup and when finished, starting it again. * Suspend mode - provides a little bit lower consistency, it includes suspending VM/container, then starting snapshot mode and creating backup and when finished, starting it again (from suspend). * Snapshot mode - provides lowest operation downtime, it includes snapshoting the VM/container, creating backup (live) without stopping the VM/container (usage of the guest agent is advised in this mode).

Snapshots

PVE also support creating the

snapshot Snapshot, snapshots or snap shot may refer to: * Snapshot (photography), a photograph taken without preparation Computing * Snapshot (computer storage), the state of a system at a particular point in time * Snapshot (file format) or SNP, a file ...

of the VM/container in time. You can create many snapshots (for example before updating the VM/container) and switch back to any one of them.

Integrated firewall

PVE includes integrated

firewall Firewall may refer to: * Firewall (computing), a technological barrier designed to prevent unauthorized or unwanted communications between computer networks or hosts * Firewall (construction), a barrier inside a building, designed to limit the spr ...

in three levels: * Cluster wide. * Per PVE node (host/hypervisor). * Per VM/Linux container (guests).

Known limitations

In Proxmox VE 7.2 (May 2022) there are known limitations: * Host (physical server) system maximum RAM: 12 TB * Host (physical server) system maximum CPU core count: 768 logical CPU cores * Number of hosts (physical servers) in

cluster: 32+ (official, but more is possible). * Maximum number of bridges per host (physical server): 4094. ** Maximum number of logical networks when using VLANs: 4094. ** Maximum number of logical networks when using VxLANs (with SDN): 16 777 216. ** Maximum number of logical networks when using QinQ (so called Stacked VLAN): 16 777 216.

References

External links

*
Project wikiProject HOWTOsProject ForumProject Roadmap
- features by the version of the Proxmox VE
Project Bug TrackerProject API
{{Authority control 2008 software Debian-based distributions Free software programmed in Perl Free virtualization software Linux distributions Perl software Software using the GNU AGPL license

History

Features

VM emulated/paravirtualized hardware

Supported VM/Linux container operating systems

Central management

Network model

Storage model

Storage replication

High-availability cluster

VM/containers migration

Clones and templates

Virtual appliances

Data backup

Snapshots

Integrated firewall

Known limitations

See also

References

External links